Image processing apparatus and method

ABSTRACT

A computer displays an image of an object which has been created by photographing by a photographing portion in a setting mode. Further, height threshold-value data is received according to an input from outside through a key board. Based on height information detected from the image created by photographing and based on the received threshold-value data, a partial image having height information specified by the threshold-value data is extracted from the image created by photographing and then is displayed. In a driving mode, the computer displays an image acquired by photographing the object, and extracts a partial image having height information specified by the threshold-value data, based on the height information detected from the image created by photographing and the threshold-value data preliminarily-received in the setting mode. The image data extracted in the driving mode is utilized for inspections of defects in the object.

This application is based on Japanese Patent Application No. 2009-060888filed with the Japan Patent Office on Mar. 13, 2009, the entire contentof which is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to an image processing apparatus andmethod which identify and extract an area having preliminarily-setheight information from acquired images.

2. Related Art

In manufacturing scenes, automatization has been advanced, in view ofpower saving and higher efficiency. In order to realize theautomatization, a number of sensors have been employed. Among suchsensors, there have been frequently employed image sensors capable ofphotographing works (target objects to be inspected) and processing theimages resulted from the photographing, in order to perform inspectionsof defects in these works.

Many image sensors are constituted by a photographing portion forphotographing works and an image processing portion for processing theimages resulted from photographing and are adapted to performdetermination as to whether or not the images resulted fromphotographing include an area having a predetermined shape, and thelike. As work detect inspections using such image processing, there areinspections for defects in works placed on pallets. Conventionally,works have been stopped at the same position in the field of view of acamera as a photographing portion, during transferring the works on amanufacture line. In order to detect work images from images resultedfrom the photographing, a measurement area is provided at apredetermined position in the field of view, and the size of themeasurement area is set to be coincident with the size of works. Thepositional deviations of works in the images resulted from thephotographing are corrected such that they are coincident with themeasurement area and, thereafter, an inspection processing is started.The correction of their positional deviations is attained by applyingsearching and edge processing to the images resulted from thephotographing.

As other methods for detecting work images from images created byphotographing, there has been suggested processing for cuttingbackgrounds using color information. As an example of this processing,the present applicant has suggested a technique described in JapaneseUnexamined Patent Publication No. 2008-15706. This technique is capableof extracting only work images (cutting backgrounds) from images createdby photographing, using color information about works.

SUMMARY

With the detection of work images using the aforementioned measurementarea, if there exists, on a pallet, a background (such as a shockabsorption member) similar to works, even by using searching and edgeprocessing, it is impossible to accurately perform search measurementand edge position measurement due to the influence of the background.This increases the difficulty of detecting the amount of positionaldeviation, thereby making it impossible to accurately correct thepositional deviation. As a result, it is necessary to take longer timefor work defect inspections and, also, it is hard to provide highdetection accuracy.

Further, with the detection of work images according to the backgroundcutting processing using the aforementioned color information, it ishard to perform processing for cutting only backgrounds, in cases wherebackgrounds and works have the same color. Further, due to the use ofcolor information, the detection tends to be influenced by externalenvironments such as the condition of illumination.

Therefore, it is an object of the present invention to provide an imageprocessing apparatus and method which are capable of detecting images ofobjects from images created by photographing, without being subjected torestrains on the placement of target objects to be detected and thecolor of the background, and without being influenced by externalenvironments such as the condition of illumination.

In accordance with one aspect of the present invention, an imageprocessing apparatus includes: a display portion; a height input portionwhich inputs height information about a target object; a setting portionwhich sets data for image processing in a setting mode; and a drivingprocessing portion which receives an input image and processes the inputimage in the driving mode based on data which has been preliminarily setby the setting portion.

The setting portion includes a set-image display portion which displays,on the display portion, a first image acquired by photographing thetarget object in the setting mode, a threshold-value reception portionwhich receives threshold-value data about the height, according toinputs from the outside, and a first detection portion which detects,from the first image, a first partial image having height informationspecified by the threshold-value data, based on the height informationinputted by the height input portion in the setting mode, and based onthe threshold-value data received by the threshold-value receptionportion.

The driving processing portion includes a driving-image display portionwhich receives a second image acquired by photographing the targetobject in the driving mode and displays the photographed target objecton the display portion, and a second detection portion which detects,from the second image, a second partial image having height informationspecified by the threshold-value data, based on the height informationinputted by the height input portion in the driving mode, and based onthe threshold-value data received by the threshold-value receptionportion in the setting mode.

Preferably, the setting portion displays the first partial imagedetected by the first detection portion on a screen of the displayportion in the setting mode, such that a background image for the firstpartial image on the screen is displayed in a predetermined color.

According to this aspect, in the setting mode, the user can check thedetected first partial image. If the user determines that a desiredimage has not been detected, he or she can make setting of thethreshold-value data, again, through inputting from the outside.

Preferably, the predetermined color of the background image can bechanged.

Preferably, the height information indicates the heights of the targetobject in correspondence with respective portions of an image of thetarget object.

Preferably, the height information indicates the heights of the targetobject in correspondence with respective portions of an image of thetarget object.

Preferably, the setting portion further includes a cross-sectional shapedisplay portion which detects the height information about the portionof the first image which corresponds to a specified position therein,based on the height information inputted by the height input portion inthe setting mode and displays, along with the first image, a first lineindicative of the cross-sectional shape of the target object whichcorresponds to the specified position, based on the detected heightinformation.

Preferably, the values of height information corresponding to thecross-sectional shape indicated by the first line are displayed inassociation with or near the first line. Preferably, it is possible tochange over between display and non-display of the values of heightinformation.

Preferably, the first image corresponds to a plane with two-dimensionalcoordinates which is defined by a first axis and a second axisorthogonal to each other, and the specified position is specified bycoordinates through which second lines parallel with the first axis andthe second axis are passed.

Preferably, a third line indicating height information specified by thethreshold-value data received by the threshold-value reception portionis displayed along with the first image, in association with the firstline.

Preferably, the values of height information is displayed in associationwith or near the third line. Preferably, it is possible to change overbetween display and non-display of the values of height information.

Preferably, the inputs from the outside include an area specificationinput which specifies a partial area in the first image displayed by theset-image display portion, and the height threshold-value data isspecified by the height information about the portion corresponding tothe specified area, out of the height information inputted by the heightinput portion.

Preferably, the inputs from the outside include a point specificationinput which specifies a plurality of points in the first image displayedby the set-image display portion, and the height threshold-value data isspecified by the height of a plane passing through the portionscorresponding to the specified plurality of points, out of the heightinformation inputted by the height input portion.

Preferably, the plane includes an upper limit plane and a lower limitplane corresponding to an upper limit and a lower limit of the thresholddata, the plurality of points include specified points for the upperlimit plane and the lower limit planes, the height threshold-value datais specified by the height information about the upper limit plane andthe lower limit plane, and the second detection portion detects thesecond partial image having the height information which is equal to orless than the height information about the upper limit plane and isequal to or more than the height information about the lower limitplane, based on the height information inputted by the height inputportion in the driving mode, and based on the height information aboutthe upper limit plane and the lower limit plane which is specified bythe threshold-value data received by the threshold-value receptionportion in the setting mode.

Preferably, the height information about the upper limit plane and thelower limit plane can be changed according to inputs from the outside.

Preferably, the height information about the target object incorrespondence with the respective portions indicate tone values of animage in correspondence with the portions, the threshold-value receptionportion includes a density-plane creation portion which detects portionsof the first image which have a larger degree of density change than adegree of density change in a peripheral area, based on the tone valuesof the respective portions and the inputs from the outside and, further,creates a curved plane passing through the detected portions, and thethreshold-value reception portion includes a plane threshold-valuereception portion which receives the threshold-value data based on theheight information corresponding to the portions through which thecurved plane passes.

Preferably, the threshold-value data received by the planethreshold-value reception portion indicates the height informationcorresponding to the portions through which the curved plane passesafter it has been updated based on an offset value.

Preferably, the offset value can be changed according to inputs from theoutside.

Preferably, a size of the peripheral area can be changed according toinputs from the outside.

In accordance with another aspect of the present invention, an imageprocessing method includes a setting step for setting data for imageprocessing in a setting mode; and a driving processing step forreceiving an input image and processing the input image based on dataset in the setting step.

The setting step includes a set-image display step for displaying afirst image acquired by photographing a target object in the settingmode, a threshold-value reception step for receiving threshold-valuedata about the height, according to inputs from outside, and a firstdetection step for detecting, from the first image, a first partialimage having height information specified by the threshold-value data,based on the height information about the target object which has beeninputted in the setting mode, and based on the threshold-value datareceived in the threshold-value reception step.

The driving processing step includes a driving-image display step forreceiving a second image acquired by photographing the target object inthe driving mode and displaying the photographed target object, and asecond detection step for detecting, from the second image, a secondpartial image having height information specified by the threshold-valuedata, based on the height information about the target object which hasbeen inputted in the driving mode, and based on the threshold-value datareceived in the threshold-value reception step in the setting mode.

With the present invention, a partial image corresponding to heightthreshold-value data according to inputs from the outside is detectedfrom an image of the target object which has been created byphotographing, based on height information about the target object. Thisenables detecting the partial image from the image created byphotographing, without being subjected to restrains on the color and thepattern of the background for the target object, and without beinginfluenced by changes of external environments such as the condition ofillumination.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view illustrating the entire structure of an imageprocessing apparatus according to an embodiment of the presentinvention;

FIG. 2 is a schematic view illustrating the hardware structure of acomputer which incorporates the image processing apparatus according tothe embodiment of the present invention;

FIGS. 3A to D are views illustrating height threshold-valuespecification methods according to an embodiment of the presentinvention;

FIG. 4 is a schematic flow chart of overall processing according to theembodiment of the present invention;

FIG. 5 is a schematic flow chart of setting processing according to anembodiment of the present invention;

FIG. 6 is a view illustrating selection of a display image according tothe embodiment of the present invention;

FIGS. 7A to C are views illustrating exemplary candidate images to beselected as a display image according to the embodiment of the presentinvention;

FIG. 8 is a view illustrating procedures for creating an LUT accordingto the embodiment of the present invention;

FIG. 9 is a view illustrating display of a profile in an image,according to the embodiment of the present invention;

FIG. 10 is a processing flow chart for parameter setting according to apoint specification method according to the embodiment of the presentinvention;

FIG. 11 is a view illustrating an exemplary screen displayed in theprocessing flow chart in FIG. 10;

FIG. 12 is a view illustrating another exemplary screen displayed in theprocessing flow chart in FIG. 10;

FIG. 13 is a view illustrating still another exemplary screen displayedin the processing flow chart in FIG. 10;

FIG. 14 is a view illustrating still another exemplary screen displayedin the processing flow chart in FIG. 10;

FIG. 15 is a processing flow chart for parameter setting according to astatic plane specification method according to the present embodiment;

FIG. 16 is a view illustrating an exemplary screen displayed in theprocessing flow chart in FIG. 15;

FIG. 17 is a view illustrating another exemplary screen displayed in theprocessing flow chart in FIG. 15;

FIG. 18 is a processing flow chart for parameter setting according to adynamic plane specification method according to the embodiment of thepresent invention;

FIG. 19 is a view illustrating an exemplary screen displayed in theprocessing flow chart in FIG. 18;

FIG. 20 is a view illustrating another exemplary screen displayed in theprocessing flow chart in FIG. 18;

FIG. 21 is a processing flow chart for parameter setting according to adynamic curved-plane specification method according to the embodiment ofthe present invention;

FIG. 22 is a view illustrating an exemplary screen displayed in theprocessing flow chart in FIG. 21;

FIGS. 23 A to C are views illustrating the concept of the dynamiccurved-plane specification according to the embodiment of the presentinvention;

FIG. 24 is a view illustrating a filter which is applied to the algorismof the dynamic curved-plane specification according to an embodiment ofthe present invention;

FIG. 25 is a flow chart of driving processing according to an embodimentof the present invention; and

FIG. 26 is a view illustrating an exemplary screen displayed in theprocessing flow chart in FIG. 25.

DETAILED DESCRIPTION

Embodiments of the present invention will be described in detail withreference to the drawings. Further, like reference characters refer tothe same or corresponding parts throughout the drawings, and descriptionthereof will not be repeated.

(The Entire Apparatus Structure)

Referring to FIG. 1, an image processing apparatus 1 according to anembodiment of the present invention is capable of performing opticalprocessing on target objects to be inspected (which may be referred toas “works” in some cases, hereinafter), by being incorporated in aproduction line, representatively. This optical processing includesdetection of defects in work surfaces.

As an example, in the present embodiment, pallets 14 carrying worksillustrated by diagonal lines are transferred by a transfer mechanism 16such as a belt conveyer, and the transferred pallets 14 are successivelyphotographed by a photographing portion 18. Image data (which will bereferred to as “input images” in some cases, hereinafter) is transmittedto a computer 1 which functions as an image processing apparatus.Further, such input images may be either color images or monochromeimages (gray scale) images.

An optoelectronic sensor placed at the opposite ends of the transfermechanism 16 detects the pallets 14 having reached to the photographingrange of the photographing portion 18. More specifically, theoptoelectronic sensor is constituted by a light receiving portion 15 aand a light projection portion 15 b placed on the same optical axis,such that the light projection portion 15 b emits light, and the lightreceiving portion 15 a detects the emitted light which has beeninterrupted by a pallet 14 to detect the pallet 14 having reached to thephotographing range of the photographing portion 18.

If the aforementioned optoelectronic sensor detects a pallet 14 havingreached thereto, the photographing portion 18, in response thereto,starts photographing the pallet 14 including works. Also, it is possibleto start detection of defects in works and other inspections andmeasurements at the timing when a pallet 14 having reached thereto isdetected, while continuously performing photographing.

The photographing portion 18 includes a first camera portion 2A and asecond camera portion 2B which will be described later.

A CD-ROM (Compact Disk-Read Only Memory) 12 is detachably mounted to thecomputer portion 1, and the computer 1 incorporates the function ofaccessing the mounted CD-ROM. Further, the computer portion 1 includes akey board 6A and a mouse 6B. Further, the computer 1 executes a programpreliminarily stored therein for providing a method for supportingsetting of parameters for use in image processing for detecting works tobe inspected in images according to the present embodiment.

(The Hardware Structure)

Referring to FIG. 2, the computer 1 includes a CPU (Central ProcessingUnit) 4, a main storage portion 8 including a ROM (Read Only Memory) anda RAM (Random Access Memory), a display processing portion 9 whichperforms processing for information display relating to the displayportion 3, an external I/F (Interface) 10 for connection to externalcommunication networks and the like, an input portion 6 for receivinginput data from the key board 6A and the mouse 6B, a reading portion 11which accesses a recording medium such as the CD-ROM 12 and reading datatherefrom, a secondary storage portion 5 such as a hard disk, and aphotographing portion I/F 7 for inputting image data created byphotographing by the photographing portion 18. These portions areconnected to one another through buses 13.

The photographing portion 18 includes the first camera portion 2A andthe second camera portion 2B. If the aforementioned optoelectronicsensor detects a pallet 14 carrying works which has reached thereto, thefirst camera portion 2A, in response thereto, starts photographing thepallet 14 including the works.

As an example, the first camera portion 2A is structured to include animage pickup device partitioned into plural pixels, such as a CCD(Charge Coupled Device) or a CIS (Contact Image sensor), in addition tooptical systems such as lenses. The second camera portion 2B is providedfor performing photographing for distance images, which will bedescribed later.

The image processing apparatus according to the present embodiment isrealized by causing the CPU 4 to execute programs, using computerhardware, such as the main storage portion 8, the secondary storageportion 5 and the CD-ROM 12. In general, these programs are stored in arecording medium, such as the CD-ROM 12, and are distributed throughvarious communication networks, such as the Internet. Further, theseprograms are read from the recording medium such as the CD-ROM 12 and istemporarily stored in the secondary storage portion 5 which is aninternal storage device. Further, these programs are read from thesecondary storage portion 5 into the main storage portion 8 and then areexecuted by the CPU 4.

The CPU 4 is a calculation processing portion which executes varioustypes of calculations, by sequentially executing programmed commands.

The photographing portion I/F 7 intermediates between the computer 1 andthe photographing portion 18 for data communication therebetween. Thephotographing portion I/F 7 preferably includes an image buffer. Theimage buffer temporarily accumulates input image data resulted fromphotographing which have been successively transferred through thephotographing I/F 7. Further, if input image data resulted from onephotographing is accumulated therein, the image buffer transfers theaccumulated data to the main storage portion 8 or the secondary storageportion 5.

The external I/F 10 is a device which intermediates between the computer1 and external devices for data communication therebetween. The externalI/F 10 receives detection signals from the aforementioned optoelectronicsensor and the like and transfers the received signals to the CPU 4.

The secondary storage portion 5 is a nonvolatile storage device whichstores input image data and programs to be executed by the CPU 4.

The display portion 3 which is connected to the computer 1 is fordisplaying information (including images) outputted from the CPU 4 andis constituted by an LCD (Liquid Crystal Display), a CRT (Cathode RayTube) or the like.

The mouse 6B receives commands from the user, according to clickingoperations, sliding operations and other operations. The key board 6Areceives commands from the user, according to inputs to keys therein.

Further, another output device, such as a printer, can be connected tothe computer 1, as required.

The CPU 4 displays screens on the display portion 3 in the imageprocessing apparatus according to the present embodiment, in cooperationwith a graphic board, which is not illustrated. The display of suchscreens is realized by a GUI (Graphical User Interface) programsincorporated as a portion of an OS (Operating System), and the GUIprovides an environment which enables the user to make various usersettings, by using a cursor on the screen which is operated through thekey board 6A and the mouse 6B.

(Height Threshold-Value Specification Method)

The image processing apparatus according to the present embodiment isadapted to determine the absolute distance and a relative distance toworks and creates a distance image which will be described later, inorder to detect a work image area from an input image.

An arbitrary height range is specified in the created distance image,and only the portion corresponding to the specified range is extractedfrom the input image for detecting an image of interest to be subjectedto defect inspections.

In the present embodiment, such a distance image is an image indicatingheight information exhibited by an input image with a gray scale.Namely, the luminance levels at the respective pixels in the input imageare varied depending on the distance from the photographing portion 18(whether it is far or close) to the target object to be photographedand, therefore, height information corresponding to the distance to thetarget object in correspondence with the respective pixels is detectedbased on the difference of the luminance levels. Further, the detectedheight information is converted into tone information according topredetermined procedures, and an image is created according to the toneinformation resulted from the conversion, thereby detecting a distanceimage.

As a distance determination method, there is a shape-from-focus methodwhich overlaps images which are brought into focus to create an imageand further determines the distance to target object from the focallength of the created image. Further, there is a TOF (Time Of Flight)method which detects the distance to a target object, based on the lightflight time from emission of light from a light source until the lightis reflected by a target object surface and reaches a sensor. Further,there are a method which applies a laser as a line beam to a targetobject and detects the distance from reflected light components, and astereo method using two cameras. In the present embodiment, such astereo camera using two cameras is employed for distance determination,but the distance determination method is not limited thereto and can beother methods.

Such a distance determination method using a stereo camera is wellknown. In brief, plural cameras are caused to photograph a target objectat the same time in plural different directions, so that informationabout the target object in the depthwise direction can be recorded. Inthe present embodiment, the second camera portion 2B includes twocameras placed at different positions. The two cameras photograph asingle target object (a pallet 14 or the like), which induces a parallaxbetween the images created by the photographing by the two cameras (thecameras placed at different positions in the left and right sides). TheCPU 4 calculates the distance from the cameras to the target object,based on the images created by the photographing, based on thisparallax, according to a predetermined calculation equation, to obtaininformation about the distance to the target object. Further, the CPU 4displaces the two cameras, little by little, and determines thedistances from the cameras at respective points above the target objectsurface to these points. This enables detection of distance informationin correspondence with portions corresponding to the respective pointsabove the target object surface or in correspondence with the respectivepixels. Based on the height information which is the detected distanceinformation, the CPU 4 searches an LUT (Look Up Table), further readscorresponding gray scale values (tone information) from the LUT andcreates a distance image based on the read tone information.

In the detected distance image as described above, FIGS. 3(A) to (D)schematically illustrate methods for specifying threshold value (heightthreshold values) for use in detecting an image area to be extracted asa target object to be inspected. When the target object is certainlyhorizontal with respect to a reference plane, it is possible to specifyheight threshold values, according to a “point specification method”(FIG. 3A). When the target object is certainly placed with a constantinclination with respect to a reference plane, it is possible to specifyheight threshold values, according to a “static-plane specificationmethod” (FIG. 3B). When the inclination of the target object withrespect to a reference plane is varied, it is possible to specify heightthreshold values, according to a “dynamic plane specification method”(FIG. 3C). When the target object has an unstable shape, it is possibleto specify height threshold values, according to a “dynamic curved-planespecification method” (FIG. 3D).

In this case, the reference plane refers to a plane orthogonal to theoptical axes of the cameras in the photographing portion 18 or atransferring surface of the conveyer. Thus, the reference plane is aplane indicating an inclination with respect to a predeterminedcoordinate system, based on the predetermined coordinate system.

(Schematic Flow Chart)

Referring to FIG. 4, the image processing apparatus according to thepresent embodiment has a parameter setting mode, a driving mode and aninspection mode, as operation modes. It is possible to change over amongthese operation modes, according to inputs from the outside.

The image processing apparatus can perform overall processing includingprocessing for setting parameters for use in extracting (detecting) animage of interest from a distance image in the parameter setting mode(step S3), driving processing for detecting an image to be inspectedfrom an input image using preset parameters in the driving mode (stepS5), and processing for inspections for defects in works using the imagedetected through the driving processing in the inspection mode (stepS7). Further, in FIG. 4, steps S3 to S7 are illustrated as beingcontinuously executed, but the order of execution of the respectiveprocessing is not limited thereto, provided that necessary parametervalues or image data have been detected when the processing in the earlystages illustrated in FIG. 4 is completed.

(Schematic Processing for Setting Parameters)

Referring to FIG. 5, the parameter setting processing (step3) in FIG. 4includes inputting of selection of a display image by the user (stepS33) at first, selectively determining a height threshold-valuespecification method (any one of the methods in FIG. 3(A) to (D)) (stepS35), processing for setting parameter values according to thedetermined height threshold-value specification method (step S37), andprocessing for displaying an image extracted from the display imageselected in step33 (step S39). Hereinafter, the processing in therespective steps will be described.

Selection of a display image in step33 in FIG. 5 will be described, withreference to FIG. 6 and FIGS. 7(A) to (C).

In step33 in FIG. 5, a screen in FIG. 6 is displayed on the displayportion 3. In the screen in FIG. 6, there are displayed an image area301 for displaying an image, a “display changeover” tab 306, a “displaysetting” area 300, an “OK” button 400 which is operated for generating acommand for storage of set parameters, and a “cancel” button 401 whichis operated for generating a command for cancel of operations.

The “Display Setting” area 300 is an area which receives inputs forsetting a type of images to be displayed in the image area 301. In thiscase, it is possible to select any of “Measurement Image” which indicateinput images resulted from photographing by the first camera portion 2A,“Distance Image” and “Super-deep Image”. In FIG. 6, “Input Image” isbeing selected, and an input image is being displayed in the image area301. The “Display Changeover” tab 306 is selectively operated forchanging over between moving images and static images, with respect tothe image displayed in the image area 301.

FIGS. 7(A) to (C) illustrate images displayed in the image area 301 inFIG. 6. FIG. 7(A) illustrates an input image which is an image createdby photographing by the first camera portion 2A, FIG. 7(B) illustrates asuper-deep image, and FIG. 7(C) illustrates a distance image. Such asuper-deep image is an image which is brought into focus at all pointstherein and is displayed for providing, for the user, a clearer imagethan the input image. Such a super-deep image can be created byphotographing a target object plural times with the first camera portion2A and overlapping these images with one another according to theshape-from-focus method. The CPU 4 creates such a super-deep image bysynthesizing input images outputted from the first camera portion 2A,according to the shape-from-focus method. The created super-deep imageis stored in the main storage portion 8, and the data of the super-deepimage is read from the main storage portion 8 and then is displayed onthe display portion 3.

The CPU 4 also creates a distance image according to the aforementionedprocedures and stores the distance image in the main storage portion 8.The data of the distance image is read from the main storage portion 8and is displayed on the display portion 3.

FIG. 8 schematically illustrates the content of the look up table (LUT).The LUT stores gray scale values in association with respective heightinformation. The LUT is determined by using, as input values, heightinformation detected using the aforementioned stereo camera, andcorresponding gray scale values are read therefrom as output values forcreating a distance image using the read gray scale values. In thiscase, gray scale values (tone values) are values in the range of 0 to255, and height information has an upper limit value and a lower limitvalue corresponding to gray scale values (tone values). The distanceimage has coordinate values (X, Y, Z) in association with the respectivepixels therein. Each Z coordinate value is a coordinate along a Z axisorthogonal to the reference plane and indicates height information,while the coordinate values (X, Y) are coordinates along an X axis and aY axis which are orthogonal to the Z axis. The reference plane andimages are assumed to be planes having two-dimensional coordinates whichare defined by the X axis and the Y axis which are orthogonal to eachother.

In this case, for selection of a display image, a super-deep image (seeFIG. 7(B)) is suggested as a reference image for the input image, but“Distance-Information Superimposed Image” can be suggested, instead ofsuper-deep images. The CPU 4 detects such a “Distance-InformationSuperimposed Image”, as follows.

Namely, an input image or a super-deep image which is an image createdby photographing by the first camera portion 2A is converted into a grayscale image, based on the tone values of the respective pixels. Further,based on the tone values of the respective pixels in the gray scaleimage, the LUT is searched, and the corresponding height information isread therefrom. Further, the read height information is processedaccording to a predetermined conversion equation for calculating therespective tone values for R (red), G (green) and B (blue) correspondingto the respective pixels. Image data is created using the tone valuesfor the three primary colors (RGB) which have been determined throughthe calculations. The created image data indicates a“Distance-Information Superimposed Image”. When such a“Distance-Information Superimposed Image” is displayed on the displayportion 3, the user can check, from color differences therein, theheight information exhibited by the input image.

(The Display of Profile)

In the present embodiment, for setting parameters, along with an inputimage displayed on the image area 301 in FIG. 9, based on the heightinformation about a predetermined position in this input image, it ispossible to display a profile indicative of the cross-sectional shape ofwork which corresponds to the predetermined position, by superimposingthe profile thereon.

For displaying profile information about height information, any one offour types “horizontal and vertical”, “only horizontal”, “only vertical”and “non display” can be selected. In this case, assuming that a planeparallel with the reference plane is a plane with two-dimensionalcoordinates which is defined by the X axis and the Y axis orthogonal toeach other, “vertical” indicates the direction in which the Y axisextends, and “horizontal” indicates the direction in which the X axisextends. When selecting profile display, the user clicks a desiredposition in the image area 301 by operating the mouse 6B. This desiredposition refers to a “tap position”.

Referring to FIG. 9, if a tap position 303 is specified, the CPU 4 drawsyellow broken lines Y2 orthogonal to each other at the coordinates (X,Y) of the tap position 303 in the image area 301 by superimposing thelines Y2 on the image. In this case, it is assumed that the user haspreliminarily selected “horizontal and vertical” for profile display.

Subsequently, based on the selection of “horizontal and vertical” by theuser, and based on the height information about the distance imagestored in the main storage portion 8, the CPU 4 extracts the heightinformation about the respective pixels through which the broken linesY2 pass, further plots the values indicated by the height informationabout the corresponding pixels in the vertical (Y) direction and thehorizontal (X) direction, and connects these plotted values to oneanother with yellow solid lines Y1. The solid lines Y1 drawn to besuperimposed on the image are referred to as a profile. Since theprofile is displayed, the user can check the change of the heightinformation about the work (the height of the cross-sectional shapethereof) along the horizontal axis and the vertical axis (correspondingto the broken lines Y2) which are orthogonal to each other at the tapposition 303.

The CPU 4 draws blue broken lines B in the image area 301 for indicatingthe upper limit value and the lower limit value of the heightinformation, by superimposing the lines B on the image. These brokenlines B can provide, for the user, information about the scale (theresolution) for specification of height information by the user. Theupper limit value and the lower limit value are detected from thedistance image. The upper limit value and the lower limit value whichhave been detected are displayed near the broken lines B.

The user draws red solid lines R which are vertical and horizontal linesin the image area 301 by operating the mouse 6B, while referring to thebroken lines B and the upper and lower limit values, in order to specifya plane having height information which is desired to be extracted. Withrespect to the drawn solid lines R, the CPU 4 calculates the numericalvalues of the heights according to the scale indicated by the brokenlines B, and then displays the calculated values near the solid lines R.This enables the user to check the height information about the desiredextraction plane, through the numerical values.

In FIG. 9, with respect to the broken lines B corresponding to the scaleand the solid lines R indicative of the height information about theextraction plane, there are displayed the numerical values of the heightinformation near the lines, but the display thereof can be eliminated.Further, it is possible to enable changeover between display andnon-display of numerical values, according to inputs of the selection bythe user.

(The Specification of Height Threshold Values)

When a desired method is selected from the height threshold-valuespecification methods illustrated in FIGS. 3(A) to (D), the CPU 4displays a list of these methods on the display portion 3, and the userinputs a selection of a desired method out of the list, by operating thekey board 6A or the mouse 6B. The CPU 4 executes a program according toa processing flow chart for the height threshold-value specificationmethod which has been selectively specified.

(The Point Specification Method)

According to the flow chart in FIG. 10, with reference to exemplarydisplayed screens in FIG. 11 and FIG. 12, there will be described amethod for specifying height threshold values relating to an extractionplane according to the point specification method.

FIG. 11 illustrates a screen for setting height threshold valuesaccording to the point specification method, at a state where a profileis displayed. In an area 303 in FIG. 11, there are displayed sliders forspecifying height threshold values for the upper limit and the lowerlimit. These sliders are each constituted by a bar having scale marksfor height information and a button 304 which moves on the bar in thedirection of an arrow in the figure according to dragging operations onthe mouse 6B. The scales of the scale marks on the bars are determinedaccording to the upper and lower limit values of the height informationwhich are indicated by the blue broken lines B. If the user moves thebuttons 304 on the bars to positions illustrated in FIG. 12, forexample, by sliding the buttons 304 through dragging operations, the CPU4 reads the positions on the bars after they have been moved, furthercalculates the values of the height information which correspond to theread positions as the upper and lower limit values, and displays thecalculated upper and lower limit values in boxes 305.

In the boxes 305 in FIG. 12, there are displayed the upper limit value(=100) and the lower limit value (=50) out of the height thresholdvalues specified through the slider operations. In FIG. 12, as an imagedisplayed in the image area 301, there is displayed a partial image inthe area corresponding to the specified threshold values, which has beenextracted from the display image selected by the user in step33. For thebackground image therefor, the user can specify a desired display color.The display color is preferably such a color as to make the extractedimage conspicuous. The image displayed in the image area 301 can bechanged over between only the partial image (the extracted image)corresponding to the specified height threshold value as in FIG. 12 andthe image which has not been subjected to extraction (the entire imageselected in step33) by operating a “Front/Rear Display Changeover” tab322.

Further, if the user updates the upper and lower limit values by slidingthe buttons 304 on the bars through a dragging operation in the screenin FIG. 12, the CPU 4 performs processing for treating this draggingoperation as an interrupt. Thus, in response to this interrupt, the CPU4 performs processing for detecting an image corresponding to theupdated upper and lower limit values from the image specified in step33and, also, for updating the display in the image area 301 in FIG. 12using this detected image.

Referring to FIG. 10, in operation, the CPU 4 creates an LUT asillustrated in FIG. 8, according to information inputted by the userthrough the key board 6A or the mouse 6B (step T1).

Subsequently, the CPU 4 creates and stores a distance image, accordingto the aforementioned procedures, based on image information having aparallax in the stereo camera which has been inputted from the secondcamera portion 2B (step T3).

Subsequently, the CPU 4 displays, in the image area 301, the displayimage selected by the user in step33 and a profile according to user'soperations (step T5). Thus, the selected display image and the profileindicated by yellow solid lines Y2 according to the specified tapposition 303 are displayed in the image area 301 in FIG. 9. Alongtherewith, blue broken lines B indicative of a scale are displayedtherein.

Subsequently, the CPU 4 determines whether or not the “OK” button 400has been clicked (step T7). When the user completes the parametersetting according to the point specification method, he or she clicksthe “OK” button 400. When it has been clicked, the CPU 4 stores the setdata in the secondary storage portion 5. At this time, as shown in theFigure, the secondary storage portion 5 has stored information includingthe LUT data 51 created in step T1, the data 52 of the display imagewhich is to be displayed in the image display area 301, and the data 53of the threshold values (the upper and lower limit values) of the heightinformation about the extraction plane set by the user through slideroperations (ST15), as illustrated in the figure.

Although in step T15, the data is stored in the secondary storageportion 5, it can be stored in the main storage portion 8.

After the completion of the data storage, the processing returns to theoriginal processing (the processing in FIG. 5).

If it is determined that the “OK” button 400 has not been clicked (No instep T7), it is determined whether or not the “Cancel” button 401 hasbeen operated (step T9). When the user cancels the parameter settingoperation, he or she operates the “Cancel” button 401. Therefore, if theCPU 4 detects the “Cancel” button 401 having been operated (Yes in stepT9), the CPU 4 stores none of the set data and returns the processing tothe original processing.

If it is determined that the “Cancel” button 401 has not been operated(No in step T9), the CPU 4 determines whether or not a rectangular shape307 has been drawn through user's operations at an arbitrary position onthe display image in the image area 301 (see FIG. 13).

If it is determined that a rectangular shape 307 has been drawn (Yes instep T11), the maximum value and the minimum value of the heightsindicated by the height information about the image in this rectangularshape 307 are detected from the corresponding distance image, based onthe coordinate values in the area in the rectangular shape 307 (stepT17). Based on the detected maximum and minimum values, the CPU 4 slides(moves) the button 304 on the bar relating to the upper limit value inthe area 303 in FIG. 13 to the position corresponding to the detectedmaximum value +1 and, similarly, slides (moves) the button 304 on thebar relating to the lower limit value to the position corresponding tothe detected minimum value −1 (see FIG. 14).

Subsequently, the CPU 4 updates the values 305 of the upper and lowerlimit values out of the height threshold values in the area 303 to theupper and lower limit values specified by the buttons 304 which havebeen moved (step T21).

Subsequently, the CPU 4 draws solid lines R in the image area 301according to the values 305 set in step T21, further performs “ImageExtraction” based on the values 305, and displays the detected extractedimage (step T23). Thereafter, the processing returns to step T7.

In this case, “Image Extraction” refers to searching a distance image,further detecting the coordinates (X, Y, Z) of the pixels having Zcoordinates values corresponding to the height information having thevalues 305 set in step T21, further searching the data of the displayimage selected in step33 based on the detected coordinates and readingonly the image data constituted by the pixels corresponding to thesecoordinates. The read image data is displayed, so that the extractedimage in FIG. 12 is displayed.

On the other hand, if it is determined that no rectangular shape 307 hasbeen drawn (No in step T11), the CPU 4 determines whether or not thebuttons 304 on the bars in the area 303 have been slid (moved) (see FIG.11). If the user has moved the buttons 304 to positions as illustratedin FIG. 12, for example, through sliding operations, the CPU 4 displays,as the values 305, the upper and lower limit values out of the heightthreshold values which correspond to the positions to which they havebeen moved (see FIG. 12).

As described above, in setting height threshold values according to thepoint specification method, the user can make settings by performingsliding operations on the buttons 304 on the bars. Also, the user canset desired height threshold values, by drawing a rectangular shape 307in the display area 301 for automatically sliding and moving the buttons304 according to the upper and lower limit values detected in the areain the rectangular shape 307. Further, both the methods can be employedin combination with each other for making settings.

(Static Plane Specification Method)

There will be described the static plane specification method withreference to exemplary displayed screens in FIG. 16 and FIG. 17,according to a flow chart in FIG. 15.

In the screen in FIG. 16, in order to determine a plane parallel with areference plane for extracting an image based on height information, theuser specifies points 310 at least at three positions in the image area301, by operating the mouse 6B. It is necessary only that three or morepoints 310 are specified in order to define a plane, and if three ormore points are specified, it is possible to freely add and eliminatethem.

The CPU 4 detects the coordinates (X, Y) of the specified points 310 inthe image area 301 and searches for data of a distance imagecorresponding to the image displayed in the image area 301 (the imageselected in step33), based on the detected coordinates (X, Y). As aresult of the searching, the values of the coordinates (X, Y, Z)corresponding to the specified points 310 can be read from the distanceimage data. The CPU 4 displays the values of the coordinates (X, Y, Z)corresponding to the respective specified points 310, in a coordinatedisplay area 308 in the display portion 3.

After specifying the points 301, if the user clicks a “Plane Setting”tab 309 for selecting this, the CPU 4 creates a plane passing throughall the specified points 310. In this case, if the CPU 4 detects thatthe user has selected the “Plane Setting” tab 309 for “Upper LimitPlane”, the CPU 4 updates the plane information about the upper limitplane, using information about the created plane. If the CPU 4 detectsthat the user has selected the “Plane Setting” tab 309 for “Lower LimitPlane”, the CPU 4 updates the plane information about the lower limitplane, using information about the created plane. In FIG. 16, theheights of the upper and lower limit planes have not been specified yetand, therefore, the red solid lines R indicating the heights of theupper and lower limit planes are displayed to be superimposed on theblue broken lines B indicating the scale values.

After the heights of planes are specified based on the values in thecoordinate display area 306, according to the plane information aboutthe lower limit plane and the upper limit plane, as described above, thescreen is updated as in FIG. 17. Referring to FIG. 17, based on thespecified height values of the upper and lower limit plane, with respectto the upper and lower limit planes extending in the horizontaldirection, the CPU 4 calculates the heights of these planes at the leftand right ends of the screen and the inclinations of these planes, whenthey are viewed in the direction toward the screen. Further, the CPU 4displays the calculated values as data of information 316, 317, 318 and315. Similarly, with respect to the upper and lower limit planesextending in the vertical direction, the CPU 4 calculates the heights ofthese planes at the upper and lower ends of the screen and theinclinations of these planes. Further, the CPU 4 displays the calculatedvalues as data of information 316, 317, 318 and 315.

In FIG. 17, it is possible to selectively change over the display ofinformation in the “Plane Information” area 311, between the display ofinformation about upper and lower limit planes extending in thehorizontal direction and the display of information about upper andlower limit planes extending in the vertical direction. In FIG. 17,“Horizontal Direction” has been specified in a box 312 and, therefore,the information 316, 317, 318 and 315 about upper and lower limit planesextending in the horizontal direction are displayed in the “PlaneInformation” area 311.

Further, by performing operations for sliding buttons 317 and 319 onbars through operations on the key board 6A or the mouse 6B, the usercan variably adjust the height values in the information 316 and 318 inthe “Plane Information” area 311 to corresponding values in conjunctionwith the sliding operation. In the image area 301, the red solid lines Rare drawn at the heights of the upper limit plane 313 and the lowerlimit plane 314, in the horizontal direction and the vertical direction,in conjunction with the settings of values in the “Plane Information”area 311.

There will be described the procedures for setting parameters accordingto “the static plane specification method” described with reference tothe screens in FIG. 16 and FIG. 17, with reference to the flow chart inFIG. 15.

In step T1 to T5, the CPU 4 creates an LUT, creates a distance image anddisplays a selected display image and a profile in the image area 301 inthe display portion 3, in the same manner as described above.

The CPU 4 determines whether or not the “OK” button 400 or the “Cancel”button 401 has been operated (step T7 and step T9). If the CPU 4determines that any of the buttons has not been operated, the CPU 4determines whether or not three or more points 310 have been specifiedin the image area 301 (step T27). If it is determined that three or morepoints have not been specified (No in step T27), the processing returnsto the processing in step T7.

If the CPU 4 determines that three or more points have been specified(Yes in step T27), the CPU 4 determines whether or not the “PlaneSetting” tab 309 for “Set Upper Limit Plane” has been operated by theuser (step T29). If the CPU 4 determines that it has not been operated(No in step T29), the processing shifts to step T31.

If the CPU 4 determines that the “Plane Setting” tab 309 for “Set UpperLimit Plane” has been operated (Yes in step T29), in step T33, thevalues of the information 316 and 318 about the horizontal direction(left) and (right) and the vertical direction (upper) and (lower) forthe upper limit plane in the “Plane Information” area 311 are updated tothe values indicated by the positions of the buttons 317 and 319 on thebars. Thereafter, the processing shifts to step T37.

If the CPU 4 determines that the “Plane Setting” tab 309 for “Set LowerLimit Plane” has been operated (Yes in step T31), in step T35, thevalues of the information 316 and 318 about the horizontal direction(left) and (right) and the vertical direction (upper) and (lower) forthe lower limit plane in the “Plane Information” area 311 are updated tothe values indicated by the positions of the buttons 317 and 319 on thebars. Thereafter, the processing shifts to step T37.

In step37, the CPU 4 displays the information 316, 317, 318 and 319 inthe “Plane Information” area 311 using the values updated in step T33and T35 and, also, draws red solid lines 313 and 314 indicating theheights and the inclinations of the upper limit plane and the lowerlimit plane in the “Plane Information” area 311. Further, the CPU 4performs “Image Extraction” of image data having height informationcorresponding to the height threshold values of the upper and lowerlimit planes, from the image data specified in step33 and, then,displays the extracted image in the image area 301.

In this case, “Image Extraction” refers to searching a distance image,further detecting the coordinates (X, Y, Z) of the pixels having Zcoordinate values corresponding to between the planes (the upper limitand the lower limit) having the height information indicated by theinformation 316, 317, 318 and 319 in the “Plane Information” area 311,further searching the data of the display image selected in step33 basedon the detected coordinates, and reading only image data constitutedonly by the pixels corresponding to these coordinates. Since the readimage data is displayed, so that the extracted image is displayed in theimage area 301.

If it is determined that the “OK” button 400 has been operated (Yes instep T7), processing for storing the set data is performed (step T15).At this time, the secondary storage portion 5 has stored informationincluding the LUT data 51, the data 52 of the display image in the imagearea 301, the height data 54 of the left ends of the upper and lowerlimit planes in the horizontal direction, the height data 55 of theright ends of the same, and the height data 54 and 57 of the upper andlower ends of the upper and lower limit planes in the verticaldirection.

(The Dynamic Plane Specification Method)

Next, there will be described the dynamic plane specification methodwith reference to displayed screens in FIG. 19 and FIG. 20, according toa flow chart in FIG. 18.

Referring to FIG. 19, the user performs an operation for specifying atleast three points 310, in order to determine a plane for imageextraction having a height threshold value, while referring to the imagein the image area 301. In this case, if three or more points 310 arespecified, it is possible to freely add and eliminate points 310. Withthe specified three or more points 310, it is possible to uniquelydetermine a plane for image extraction which passes through the pixelshaving the coordinates at these points 310.

After specifying the points 310 for specifying a plane, the user clicks“Upper Limit Plane” or “Lower Limit Plane” in the “Plane Selection” tab320, in order to set the specified plane as an upper limit plane or alower limit plane, by operating the mouse 6B. Thus, it is possible toset the specified plane as an upper limit plane or a lower limit plane.

The CPU 4 detects the coordinate positions corresponding to therespective points 301 specified by the user, further detects the valuesof the coordinates (X, Y, Z) of the corresponding pixels from a distanceimage, and displays the detected values of the coordinates (X, Y, Z) inthe “Coordinate Display” area 308.

If the user desires to correct the values of the heights at the sampling(specified) points 310 when checking the values displayed in the“Coordinate Display” area 308, the user inputs an offset value forcorrection to an “Offset Setting” area 321, by operating the key board6A. More specifically, the offset value can be specified as ±α.

Based on the respective values of the coordinates (X, Y, Z) of thespecified points 310, the CPU 4 creates a plane defined by thesecoordinates (X, Y, Z), further calculates the heights of the createdplane in the horizontal direction (the heights thereof at the left endand the right end of the screen), the heights thereof in the verticaldirection (the heights thereof at the upper end and the lower end of thescreen), and the inclination thereof, and displays the calculated valuesas information 323, 324 and 325.

In this case, when the user has specified an offset value in the “OffsetSetting” area 321, the CPU 4 recalculates the values of the information323, 324 and 325 and displays the recalculated values, based on therespective coordinates (X, Y, Z) of the specified points 310 and theoffset value specified in the “Offset Setting” area 321.

If the user operates the “Front/Rear Display Changeover” tab 322, theCPU 4 performs “Image Extraction” for an image having height informationindicating heights equal to or less than the height information aboutthe upper limit plane but equal to the height of the lower limit plane,based on the height extraction planes (the upper limit plane and thelower limit plane) set by the information 323 and 324. Further, the CPU4 displays the extracted image in the image area 301 as illustrated inFIG. 20.

In this case “Image Extraction” refers to searching a distance image,further detecting the coordinates (X, Y, Z) of the pixels having Zcoordinate values corresponding to between the planes (the upper limitand the lower limit) having the height information indicated by theinformation 323 and 324, further searching the data of the display imageselected in step33 based on the detected coordinates, and reading, fromthe data of the display image, only image data constituted only by thepixels having coordinates (X, Y) corresponding to these coordinates.Since the read image data is displayed, so that the extracted image asillustrated in FIG. 20 is displayed in the image area 301.

With reference to the flow chart in FIG. 18, there will be described thedynamic plane specification method described with reference to FIG. 19and FIG. 20.

The CPU 4 creates an LUT, further creates a distance image and displaysa display image and a profile in the image area 301 (steps T1 to T5).Subsequently, the CPU 4 determines whether or not the “OK” button 400 orthe “Cancel” button 401 has been operated (step T7 and step T9). If itis determined that any of them has been operated, the set data is stored(step T15) or the continuous processing ends.

If the CPU 4 determines, in step T7 and step T9, that any of the “OK”button 400 and the “Cancel” button 401 has not been operated, the CPU 4determines, in step T39, whether or not the plane to be set by the userhas been changed from the upper limit plane to the lower limit plane orfrom the lower limit plane to the upper limit plane, based on theinputting operations performed on the “Plane Selection” tab 320. If theCPU 4 determines that such a change has been performed (Yes in stepT39), the CPU 4 changes the set values for the plane (the upper limitplane or the lower limit plane) on which this change should bereflected, using the coordinates of the points 310 as sampling points,and the correction value inputted in the “Offset Setting” area 321 (stepT43). These set values include the values of the coordinates (X, Y, Z)of the sampling points indicated by the points 310 for the upper limitplane (or the lower limit plane) and the height correction value.Thereafter, the processing shifts to step T41.

If it is determined that the plane to be set is not changed (No in stepT39), it is determined whether or not three or more points 301 have beenspecified in the image area 301, in step T41. If it is determined thatthree or more points 301 have not been specified (No in step T41), theprocessing returns to step T7.

If it is determined that three or more points 301 have been specified(Yes in step T41), the CPU 4 determines whether or not an offset valuehas been inputted in the “Offset Setting” area 321 (step T45). If it isdetermined that an offset value has been inputted (Yes in step T45), theset values specified by the information 323 and 324 are updated based onthe offset value (step T47). Thereafter, the processing shifts to theprocessing in step T49.

In step T49, based on the set parameters (the values of the information323 and 324), as illustrated in FIG. 20, the display of the red solidlines R indicative of the upper and lower limit planes 313 and 314, thedisplay of the blue broken lines B indicative of the scale and thedisplay of the yellow solid lines Y1 indicative of the profile areupdated, in the image area 301. Further, “Image Extraction” isperformed, and the extracted image is displayed (step T49). Thereafter,the processing returns to step T7.

If the “OK” button 400 is operated in step T6 in FIG. 18, the parameterdata set by the user is stored in the secondary storage portion 5. Atthis time, the secondary storage portion 5 has stored informationincluding the LUT data 51, the type 521 of the display image in theimage area 301, the coordinate values 58 of the sampling points (thepoints 310) for the upper limit plane, the height correction value forthe upper limit plane (the inputted value in the “Offset Setting” area321) 59, the coordinate values 60 of the points 310 for the lower limitplane, and the height correction value for the lower limit plane (theinputted value in the “Offset Setting” area 321) 61.

(The Dynamic Curved-Plane Specification Method)

At first, the principle of the dynamic curved-plane specification inFIG. 23 will be described, in order to describe the “dynamiccurved-plane specification method” with reference to a displayed screenin FIG. 22, according to a flow chart in FIG. 21.

With reference to FIG. 23, the CPU 4 performs binarization processingfor setting, as white pixels, pixels at portions having tone valueshigher or lower than an average tone value of the pixels in theperipheral areas, while setting the other pixels as black pixels. Thus,the pixels having a higher degree of density variation than those of theperipheral pixels are extracted as white pixels. By combining theextracted white pixels with one another, it is possible to extract aplane constituted by portions (pixels) having larger brightness(density) variations. This extraction plane is referred to as a “dynamiccurved plane”. Such a dynamic curved plane schematically indicates theouter shape of a target object to be photographed.

In order to accurately extract a “dynamic curved plane”, pixels havinglarger brightness variations are emphasized more strongly. In order toattain this, for example, the CPU 4 applies a filter 406 as illustratedin FIG. 23(C) to the tone values of a total of 9 pixels constituted by acertain center pixel of interest and the pixels positioned adjacentthereto in the upward, downward, leftward and rightward directions, in adistance image. The center (the diagonal-line portion) of the filter 406is made coincident with the pixel of interest. The CPU 4 sums the tonevalues of the nine pixels and detects an average value based on thevalue resulted from the summation. The tone value of the pixel ofinterest is compared with the average value and, as a result of thecomparison, if the tone value of the pixel of interest is higher thanthe average value, the tone value of this pixel of interest is replacedwith the value of a while pixel. The filter 406 is applied to an inputimage while moving the pixel of interest therein, and the tone value ofthe pixel of interest is compared with the average of the tone values ofthe peripheral pixels. Thus, pixels having abrupt changes in heightinformation (distance) (namely, pixels having a maximum value) aredetected and are replaced with white pixels. The same is applied tocases where the tone value of the pixel of interest is lower than theaverage.

In the case of extracting an area having a pixel of interest which has atone value higher than the average value of the peripheral pixels,namely a bright area, as a dynamic curved plane, as illustrated in FIG.23(A), a pixel 404 at a portion having a density value 402 higher thanthe average density value 403 of the peripheral pixels, out of thedensity values 402 indicating the tone values of respective portions ofan input image, is extracted, and the extracted pixel 404 is replacedwith a white pixel.

The user can adjust (update) the value of the density value 403 using anextraction offset value 401 for changing the position of the partialimage corresponding to the extracted image 404 in the input image. Theupdated value of the density value 403 is set as an extraction thresholdvalue 400. For example, “Image Extraction” is performed on an areahaving a tone value higher than the extraction threshold value 400. Thecoordinate values of the pixels having the same tone value as theextraction threshold value 400 are plotted, and the plotted points areconnected to one another in a plane shape to create a dynamic curvedplane as a free curved plane indicated by the red solid lines R in FIG.22.

Referring to the screen in FIG. 22, if the user desires to extract onlyan image in an area having height information indicating heights largerthan those of the free curved plane of the solid lines R indicating thedynamic curved plane extracted according to the extraction condition(the filter size), the user inputs “Higher Portion” to the “ExtractionArea” tab 326 by operating the key board 6A. On the contrary, if theuser desires to extract an image having height information indicatingheights smaller than those of the dynamic curved plane of the solidlines R, the user inputs “Lower Portion” to the “Extraction Area” tab326.

In the present embodiment, it is possible to variably set the size ofthe filter 406, which is the condition for extraction of a dynamiccurved plane. The user can variably set the filter size, by specifyingthe size in a “Filter Size” tab 327. The setting of the filter size canbe made, by performing operations for sliding a button 328 on a barwhich is provided with scale marks for the filter size, as well as byinputting a numerical value in the “Filter Size” tab 327. Further, anextraction offset value 401 can be variably inputted to an “ExtractionOffset” tab 329. Instead of inputting a numerical value to the“Extraction Offset” tab 329, it is also possible to perform inputting byperforming operations for sliding a button 330 on a bar provided withscale marks for the extraction offset value 401.

FIG. 23(B) illustrates a case of extracting an area having tone valueslower than the extraction threshold value 400, namely a dark area. Inorder to provide the extraction condition in FIG. 23(B), it is possibleto set “Lower Portion” in the “Extraction Area” tab 326 in FIG. 22.

(The Dynamic Curved-Plane Extraction Algorism and Extracted Image)

The algorism for the aforementioned dynamic curved-plane extraction willbe described. In order to specify a dynamic curved plane, all the pixelsin an input image I (i, j) are set as an pixel of interest, and a filter406 with a size of 5×5 in FIG. 24 is applied to a total of 25 pixelswhich are constituted by a pixel of interest and the peripheral pixels,such that the center (the diagonal-line portion) of the filter 406 iscoincident with the pixel of interest. The CPU 4 detects the differencebetween the tone value of the pixel of interest and the tone values ofthe peripheral pixels, with respect to all the pixels in the input image(i, j), to calculate (create) a difference image DI (i, j) havingdetected differences assigned to the respective pixels in the inputimage (i, j).

$\begin{matrix}{{{DI}\left( {i,j} \right)} = {{I\left( {i,j} \right)} - \frac{\sum\limits_{l = {- 2}}^{2}{\sum\limits_{k = {- 2}}^{2}{I\left( {{i + k},{j + l}} \right)}}}{5 \times 5}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

Further, in order to perform “Image Extraction” from the input image (i,j), the CPU 4 compares the difference at each pixel in the differenceimage DI (i, j) with an offset value “offset” corresponding to theextraction offset value 410. Further, based on the result of thecomparison, the CPU 4 creates (calculates) an output image O (i, j)corresponding to the extracted image.

More specifically, when “Higher Portion” has been inputted to the“Extraction Area” tab 326, an output image O (i, j) is calculated basedon the comparison equation (Equation 2). When “Lower Portion” has beeninputted to the “Extracted Area” tab 326, an output image O (i, j) iscalculated based on the comparison equation (Equation 3).DI(i,j)>offset→O(i,j)=255DI(i,j)≦offset→O(i,j)=0  (Equation 2)DI(i,j)<−offset→O(i,j)=255DI(i,j)≧−offset→O(i,j)=0  (Equation 3)

With reference to the flow chart in FIG. 21, the processing flow chartfor the dynamic curved-plane specification method will be described.

The CPU 4 creates an LUT, creates a distance image and displays adisplay image and a profile in the image area (steps T1 to T5).Subsequently, the CPU 4 determines whether or not the “OK” button 400 orthe “Cancel” button 401 has been operated (step T7 and step T9). If itis determined that any of them has been operated, the set data is stored(step T15) or the continuous processing ends.

If the CPU 4 determines, in step T7 and step T9, that any of the “OK”button 400 and the “Cancel” button 401 has not been operated, the CPU 4determines whether or not the user has changed the settings of theto-be-extracted area of the image, based on the data inputted in the“Extracted Area” tab 326 (step T53). If the CPU 4 detects that thesettings have been changed (Yes in step T53), the set area is set as anarea to be extracted (step T59). Thereafter, the processing shifts tostep T65 which will be described later.

If it is detected that the settings have not been changed (No in stepT53), it is determined whether or not the user has changed the filtersize, in the “Filter Size” tab 327 (step T55). If it is detected thatthe filter size has been changed (Yes in step T55), the set filter sizeis set as a to-be-used size of the filter 406 (step T61). Thereafter,the processing shifts to step T65 which will be described later.

If it is detected that the setting of the filter size has not beenchanged (No in step T55), it is determined whether or not the user haschanged the extraction offset value 401 in the “Extraction Offset” tab329 (step T57). If it is detected that the extraction offset value hasbeen changed (Yes in step T57), the set extraction offset value is setas the offset value 401 to be used for image extraction as illustratedin FIG. 23 (step T63). Thereafter, the processing shifts to step T65which will be described later.

In step T65, based on the set parameters, as illustrated in FIG. 22, thedisplay of the red solid lines R indicative of the dynamic curved plane,the display of the blue broken lines B indicative of the scale, and thedisplay of the yellow solid lines Y1 indicative of the profile areupdated, in the image area 301. Further, “Image Extraction” isperformed, based on the values set by the user in the “Extraction Area”tab 326, the “Filter Size” tab 327 and the “Extraction Offset” tab 329and based on the aforementioned equation, and the extracted image isdisplayed (step T65). Thereafter, the processing returns to step T7.

If the “OK” button 400 is operated in step T7 in FIG. 21 (Yes in stepT7), the parameter data set by the user is stored in the secondarystorage portion 5. At this time, the secondary storage portion 5 hasstored information including the LUT data 51, the data 52 of the displayimage in the image area 301, the data 62 which specifies the extractionarea, the data 63 of the size of the filter 406, and the data 64indicating the extraction offset 401.

Although the aforementioned respective height threshold-valuespecification methods are adapted to actually photograph works 14through the photographing portion 18 and create a distance image basedon the created image data (refer to step T3), the acquisition path isnot limited thereto. Namely, in step T3, data of apreliminarily-prepared distance image can be acquired by inputting itfrom a network or a CD-ROM 12 through the external I/F 10 or the readingportion 11 or by temporarily storing the data in the secondary storageportion 5 and then reading it from the secondary storage portion 5.

(Driving Processing)

If parameter values are stored in the secondary storage portion 5according to the aforementioned height threshold-value specificationmethods, the CPU 4 performs processing in the driving mode forinspecting work defects using the parameter value of the secondarystorage portion 5 (step S5). The processing in the driving mode will bedescribed, with reference to a driving screen in FIG. 26 which isdisplayed in the display portion 3, according to a flow chart in FIG.25.

At first, in the event of transition to the driving mode according to acommand inputted from the outside, the CPU 4 reads the parameter valuesstored in the secondary storage portion 5 and stores the values in themain storage portion 8.

Subsequently, the CPU 4 causes the first camera portion 2A in thephotographing portion 18 to photograph pallets 14 being transferredthrough the conveyer 16. An input image which is an image created byphotographing by the first camera portion 2A is stored in the mainstorage portion 8 (step S51).

Further, the second camera portion 2B is caused to photograph thepallets 14, and a parallax image which is an image created by thephotographing is outputted therefrom. The CPU 4 detects the heightinformation about the respective pixels based on the parallax image, andcreates (detects) a distance image (a density image) based on thedetected height information and the LUT read from the main storageportion 8 (step S53). The distance image is stored in the main storageportion 8.

Subsequently, the CPU 4 reads, from the main storage portion 8, thevalues of the set parameters according to the “Height Threshold-ValueSpecification Method” preliminarily specified by the user. Further, theCPU 4 performs processing for the aforementioned “Image Extraction”according to this specified method, from the input image, based on theheight threshold values indicated by the read parameter values. Throughthis processing, an extracted image is detected and is stored in themain storage portion 8 (step S55). The extracted image is displayed inthe display portion 3 (see FIG. 26). In this case, similarly, the colorof the display of the background for the extracted image can be changed,such that the user can easily check the extracted image.

(Defect Inspection)

The CPU 4 performs defect inspection processing (step7 in FIG. 4), usingthe extracted image detected according to the aforementioned procedures.It is possible to employ, for defect inspections, well-known proceduressuch as pattern matching between model images and extracted images and,therefore, the defect inspections will not be described in detail.

It should be understood that the embodiments disclosed herein are merelyillustrative in all respects and not restrictive. The scope of thepresent invention is defined by the claims, not by the aforementioneddescription, and is intended to cover equivalent meanings to the claimsand all changes which fall within the claims.

1. An image processing apparatus comprising one or more computerscomprising: a display portion; a height input portion configured toinput height information about an object to be photographed, the heightinformation indicating the heights of the object in correspondence withrespective portions of a corresponding image of the object, and being inassociation with respective tone values of a corresponding image incorrespondence with these portions, a setting portion configured to setdata for image processing in a setting mode; and a driving processingportion configured to receive an input image and processes the inputimage based on data which has been preliminarily set by the settingportion; wherein the setting portion comprises: a set-image displayportion configured to display, on the display portion, a first imageacquired by photographing the object in the setting mode, athreshold-value reception portion configured to receive threshold-valuedata about the height, according to inputs from the outside, thethreshold-value reception portion comprising a density-plane creationportion configured to detect portions of the first image which have alarger tone value compared to an average tone value in the peripheralarea, based on the tone values of the respective portions and the inputsfrom the outside and, further, to create a curved plane passing throughthe detected portions, and further comprising a plane threshold-valuereception portion configured to receive the threshold-value data basedon the height information corresponding to the portions through whichthe curved plane passes, and a first detection portion configured todetect, from the first image, a first partial image having only heightinformation larger or smaller than height information of the curvedplane specified by the threshold-value data, based on the heightinformation inputted by the height input portion in the setting mode,and based on the threshold-value data received by the threshold-valuereception portion, and the driving processing portion comprises adriving-image display portion configured to receive a second imageacquired by photographing the object in the driving mode and configuredto display it on the display portion, and a second detection portionconfigured to detect, from the second image, a second partial imagehaving only height information larger or smaller than height informationof the curved plane specified by the threshold-value data, based on theheight information inputted by the height input portion in the drivingmode, and based on the threshold-value data received by thethreshold-value reception portion in the setting mode.
 2. The imageprocessing apparatus according to claim 1, wherein the setting portionis configured to display the first partial image detected by the firstdetection portion on a screen of the display portion in the settingmode, such that the background image for the first partial image on thescreen is displayed in a predetermined color.
 3. The image processingapparatus according to claim 1, wherein the setting portion furthercomprises a cross-sectional shape display portion configured to detectthe height information about the portion of the first image whichcorresponds to a specified position therein, based on the heightinformation inputted by the height input portion in the setting mode andto display, along with the first image, a first line indicative of thecross-sectional shape of the object which corresponds to this specifiedposition, based on the detected height information.
 4. The imageprocessing apparatus according to claim 3, wherein the first imagecorresponds to a plane with two-dimensional coordinates which is definedby a first axis and a second axis orthogonal to each other, and thespecified position is specified by the coordinates through which secondlines parallel with the first axis and the second axis are passed. 5.The image processing apparatus according to claim 3, wherein a thirdline indicating height information indicated by the threshold-value datareceived by the threshold-value reception portion is displayed alongwith the first image, in association with the first line.
 6. The imageprocessing apparatus according to claim 1, wherein the inputs from theoutside include an area specification input which specifies an area inthe first image displayed by the set-image display portion, and theheight threshold-value data is specified by the height information aboutthe portion corresponding to the specified area, out of the heightinformation inputted by the height input portion.
 7. The imageprocessing apparatus according to claim 1, wherein the inputs from theoutside include a point specification input which specifies pluralpoints in the first image displayed by the set-image display portion,and the height threshold-value data is specified by the height of aplane passing through the portions corresponding to the specified pluralpoints, out of the height information inputted by the height inputportion.
 8. The image processing apparatus according to claim 7, whereinthe plane comprises an upper limit plane and a lower limit planecorresponding to an upper limit and a lower limit of the threshold data,the plural points include specified points for the upper limit plane andthe lower limit planes, the height threshold-value data is specified bythe height information about the upper limit plane and the lower limitplane, and the second detection portion is configured to detect thesecond partial image having the height information which is equal to orless than the height information about the upper limit plane but isequal to or more than the height information about the lower limitplane, based on the height information inputted by the height inputportion, and based on the height information about the upper limit planeand the lower limit plane which is specified by the threshold-value datareceived by the threshold-value reception portion in the setting mode.9. The image processing apparatus according to claim 8, wherein theheight information about the upper limit plane and the lower limit planecan be changed according to inputs from the outside.
 10. The imageprocessing apparatus according to claim 1, wherein the threshold-valuedata received by the plane threshold-value reception portion indicatesthe height information corresponding to the portions through which thecurved plane passes after it has been updated based on an offset value.11. The image processing apparatus according to claim 10, wherein theoffset value can be changed according to inputs from the outside. 12.The image processing apparatus according to claim 1, wherein the size ofthe peripheral area can be changed according to inputs from the outside.13. An image processing method comprising the following steps using oneor more computers: a setting step for setting data for image processingin a setting mode; and a driving processing step for receiving an inputimage and processing the input image based on data set in the settingstep; wherein the setting step comprises: a set-image display step fordisplaying a first image acquired by photographing an object in thesetting mode, and a threshold-value reception step for receivingthreshold-value data about the height, according to inputs from theoutside, the height information about the object indicating the heightsof the object in correspondence with respective portions of an image ofthe object and being in association with respective tone values of acorresponding image in correspondence with these portions, thethreshold-value reception steps comprising: a density-plane creationstep for detecting portions of the first image which have a larger tonevalue compared to an average value in the peripheral area based on thetone values of the respective portions and the inputs from the outsideand, further, for creating a curved plane passing through the detectedportions, and a plane threshold-value reception portion for receivingthe threshold-value data based on the height information correspondingto the portions through which the curved plane passes, the setting stepfurther comprising: a first detection step for detecting, from the firstimage, a first partial image having only height information larger orsmaller than height information of the curved plane specified by thethreshold-value data, based on the height information about the objectwhich has been inputted in the setting mode, and based on thethreshold-value data received in the threshold-value reception step,wherein the driving processing step comprises: a driving-image displaystep for receiving a second image acquired by photographing the objectin the driving mode and displaying it, and a second detection step fordetecting, from the second image, a second partial image having onlyheight information larger or smaller than height information of thecurved plane specified by the threshold-value data, based on the heightinformation about the object which has been inputted in the drivingmode, and based on the threshold-value data received in thethreshold-value reception step in the setting mode.